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Introduction 


The U.S. Department of Education (ED) provided feedback to several states recently regarding 
the states’ ESSA plan. Among other issues, ED raised questions regarding the use of indices for 
the achievement indicator. ED expressed concerns that some of the states’ index systems 
allowed the performance of high achieving students to “mask” the performance of lower 
achieving students. There is no question that ESSA allows for index systems where additional 
points can be awarded to schools for students that score above the proficient level. Given this 
allowance, I argue that index systems offer a powerful way to incentivize continued 
improvement for students and schools without disenfranchising lower performance individuals. 
This does not negate the ESSA requirement to report the percentage of students scoring at the 
proficient level and above for all students and by student group. 


The Rationale for Index Approaches 
Index systems are designed to award points to schools (or other entities) for students scoring at 


different achievement levels. Almost all states and consortia report performance at four or five 
achievement levels. An index system would provide different points for each level of 
achievement. There is not a specific formula for how to provide points for the different levels of 
achievement. Rather, the determination of points is a policy decision designed to best represent 
the state’s values. 


Many states employed achievement indices during the No Child Left Behind (NCLB) era, but 
due to the requirements of the law, states were not allowed to provide more points than they 
awarded to a proficient score for any levels beyond proficient. In spite of this limitation, states 
still felt that index systems helped shift the focus from the “bubble students” (i.e., those who 
scored just below proficient) to a broader range of students. However, many states used more 
complete index systems (i.e., those that differentiate among all achievement levels) as part of 
their state accountability systems and now that ESSA provides similar flexibility, several states 
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have proposed such index systems as part of their state plans. To help ground the discussion, I 
present three examples of fairly typical index systems below. 


Index Points: Simple Index Points: Linear Index Points: 
Achievement Level Translation Transformation Reduced Level 4 
1 1 0 0 
2 2 50 50 
3 a 100 100 
4 4 150 125 


Figure 1. Examples of three common achievement indices. 


v The “simple translation” index system assigns the same number of points as the 
achievement level. Such an approach works well in systems where all indicators are 
converted to a 1-4 scale for reporting purposes and for straightforward aggregation 
approaches. 

v The “linear transformation” generally uses an interval-based system where all index 
points are equally spaced. The starting or ending values do not matter since a simple 
linear transformation is applied. For example, we could easily create an index system 
with values of 100, 150, 200, and 250 that would have the same meaning as the example 
shown above. With this approach, the values are typically selected to match the weight 
or influence assisgned to academic acheivement in an overall index. 

v Finally, given the concern about “masking” the performance of lower achieving students, 
the “reduced level 4” example index system uses equal intervals for the first three levels, 
but then awards the highest scoring students only one-half of an interval. As I show later, 
such an approach does not eliminate the possibility of masking the performance of lower 
achieving students. 


Masking Performance 
ED’s main concern with the use of index systems in several states was the potential that the 
points awarded to higher achieving students could mask the performance of lower achieving 
students. I start this discussion with several assertions and a definition. 

1. Masking is observed when the performance of students scoring above the proficient level 
(e.g., Level 4) obscures the performance of those students scoring below the proficient 
levels (e.g., Levels 1 and 2). This occurs when the average index value or the indicator 
score makes it appear that on average students in the school are performing well. 
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2. As long as more points are allowed to be awarded to the highest achievement level (e.g., 
Level 4), masking the performance of lower achieving students is theoretically possible 
depending on the distribution of student performance in the state. 

3. There are more and less appropriate methods for testing the degree to which a given 
index system is masking the performance of lower achieving students. The fairest test of 
masking is one where students are equally distributed among the achievement levels 
reported in the particular state. 

4. States are required to report the number and percentage of students scoring at each of the 
achievement levels on the state assessment in English language arts, mathematics, and 
science for all students and for each student group defined in ESSA. This transparent 
reporting mitigates any potential masking that might occur when the raw achievement 
results are converted to an indicator. 


I offer some examples to explain these assertions. Let’s assume an equal (flat) distribution of 
performance across achievement levels (e.g., 25% of students in each of four levels). I then 
apply the various index models to this same distribution. To compute the index values, I simply 
multiply the index values by the number (or percentage) of students scoring at that particular 
level. For the sake of these examples, assume that there are 100 students in the school, with 25 
scoring at each of the four achievement levels. We then multiply the index values by the number 
of students in each level (25) to arrive at the index points. The average index score is shown 
below for each of the three examples. This average value is fed into the accountability system, 
but it is typically transformed back to the original scale by dividing by the average number of 
students scoring at each level (in this case 25) to arrive at the “indicator value.” 


Simple Translation Linear Transformation Reduced Level 4 
Achievement Index Index Index Index Index Index 
Level Values Points Values Points Values Points 
1 dL 25 0 0 0 0 

2 2 50 50 1250 50 1250 

3 3 aD 100 2500 100 2500 

4 4 100 150 3750 125 3125 
Average Points 62.5 1875 1718.75 

Indicator Points 23 75 68.75 


Figure 2. Indicator values for three example achievement indices. 


As seen in Figure 2, the indicator points for all three examples are below the index values for 
Level 3, which means that, on average, the students in a school with these indicators points 
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would be considered performing below the proficient level. Masking would have been evident if 
the average would have been at or above the index points associated with the proficient level. 
Therefore, masking did not occur for any of these examples. 

So what would it take for masking to occur? I created a highly fictionalized example below 
(Figure 3) to illustrate potential masking. In this example, the school of 100 students has 20 
students each scoring at Levels 1 and 2, 60 students scoring at Level 4, and none at Level 3. 
Even with this extreme example, the average index value was just 3.0, which might make it seem 
like students in this school, on average, were scoring proficient. Of course, as noted in the fourth 
assertion above, the state and the school will still have to provide reports to all stakeholders so 
this type of achievement gap will be reported transparently. 


Achievement Index # of Index 
Level Values Students Points 
1 1 20 20 
2 2 20 40 
3 3 0 0 
4 4 60 240 
Average Points 75 
Indicator Points 3.0 


Figure 3. Example achievement index to illustrate masking. 


Is it possible to avoid the potential for masking if states can provide more points for Level 4 than 
they do for Level 3? Quite simply, no! Figure 4 below portrays another extreme example to 
illustrate this point. Assume the state awarded 3.1 points for students scoring at Level 4. Even 
with this miniscule increase in points compared to Level 3, a distribution such as the fictional 
distribution shown below can result in masking. 


Achievement Index # of Index 
Level Values Students Points 

1 1 2 2 

2 2 3 6 

a 3 0 0 
4 3.1 95 294.5 
Average Points 75.625 
Indicator Points 3.025 


Figure 4. Example achievement index to illustrate masking. 
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The examples presented in Figures 3 and 4 show that it is possible for masking to occur with 
extreme distributions, but this is why it makes the most sense to evaluate the masking potential 
with equal (flat) distributions (assertion #3 above). In other words, ruling out indices with any 
potential to “mask” low performance is overly broad and risks eliminating virtually any proposed 
system. 


Conclusion 
Many states are employing achievement indices as part of their ESSA-required school 
accountability systems. They are doing so to recognize achievement and changes in achievement 
along the performance distribution rather than the very narrow focus of proficient during the 
NCLB era. Such an approach to continuous improvement fits with states’ theories of action and 
is permitted by ESSA. I have argued that masking the performance of low achieving students 
may occur in cases of extremely unlikely performance distributions, but as long as the law allows 
states to provide additional points for advance performance, such masking is theoretically 
possible. Most importantly, states are required to report for all students and for each student 
group, the number and percentage of students scoring in each of the achievement levels reported 
for the state assessment system. ESSA requires both a reporting and accountability system, so 
taken together, it is essentially impossible to hide (mask) the performance of low achieving 
students as a whole or for any student group. Therefore, any achievement indices similar to the 
ones depicted in Figure 1 above should be permitted under ESSA. 
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